Q Uantized B Ack - P Ropagation : T Raining B Ina - Rized N Eural N Etworks with Q Uantized G Ra - Dients

Binarized Neural networks (BNNs) have been shown to be effective in improving network efficiency during the inference phase, after the network has been trained. However, BNNs only binarize the model parameters and activations during propagations. We show there is no inherent difficulty in training BNNs using ”Quantized BackPropagation” (QBP), in which we also quantized the error gradients and in the extreme case ternarize them. To avoid significant degradation in test accuracy, we apply stochastic ternarization and increase the number of filter maps in a each convolution layer. Using QBP has the potential to significantly improve the execution efficiency (e.g., reduce dynamic memory footprint and computational energy and speed up the training process, even after such an increase in network size.